A Family of Latent Variable Convex Relaxations for IBM Model 2

Recently, a new convex formulation of IBM Model 2 was introduced. In this paper we develop the theory further and introduce a class of convex relaxations for latent variable models which include IBM Model 2. When applied to IBM Model 2, our relaxation class subsumes the previous relaxation as a special case. As proof of concept, we study a new relaxation of IBM Model 2 which is simpler than the previous algorithm: the new relaxation relies on the use of nothing more than a multinomial EM algorithm, does not require the tuning of a learning rate, and has some favorable comparisons to IBM Model 2 in terms of F-Measure. The ideas presented could be applied to a wide range of NLP and machine learning problems. Introduction The IBM translation models (Brown et al. 1993) were the first Statistical Machine Translation (SMT) systems; their primary use in the current SMT pipeline is to seed more sophisticated models which need alignment tableaus to start their optimization procedure. Although there are several IBM Models, only IBM Model 1 can be formulated as a convex optimization problem. Other IBM Models have non-concave objective functions with multiple local optima, and solving a non-convex problem to optimality is typically a computationally intractable task. Recently, using a linearization technique, a convex relaxation of IBM Model 2 was proposed (Simion, Collins, and Stein 2013; 2014). In this work we generalize the methods introduced in (Simion, Collins, and Stein 2013) to yield a richer set of relaxation techniques. Our algorithms have comparable performance to previous work and have the potential for more applications. We make the following contributions in this paper: • We introduce a convexification method that may be applicable to a wide range of probabilistic models in NLP and machine learning. In particular, since the likelihood we are optimizing and the metric we are testing against are often not the same (e.g. for alignment tasks we want to maximize F-Measure, but F-Measure is not directly in the likelihood function), different relaxations should potentially be considered for different tasks. The crux of Copyright c © 2015, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. our approach relies on approximating the product function ∏n i=1 xi with a concave function and as a supplement we present some theoretical analysis characterizing concave functions h that approximate this function. • As a specific application, we introduce a generalized family of convex relaxations for IBM Model 2.1 Essentially, the relaxation is derived by replacing the product t(fj |ei)× d(i|j) with h(t(fj |ei), d(i|j)) where h(x1, x2) is a concave upper envelope for x1x2. We show how our results encompass the work of (Simion, Collins, and Stein 2013) as a special case. • We detail an optimization algorithm for a particularly simple relaxation of IBM Model 2. Unlike the previous work in (Simion, Collins, and Stein 2013) which relied on a exponentiated subgradient (EG) optimization method and required the tuning of a learning rate, this relaxation can be approached in a much simpler fashion and can be optimized by an EM algorithm that is very similar to the one used for IBM Models 1 and 2. We show that our method achieves a performance very similar to that of IBM Model 2 seeded with IBM 1. Notation. Throughout this paper, for any positive integer N , we use [N ] to denote {1 . . . N} and [N ]0 to denote {0 . . . N}. We denote by R+ and R++ the set of nonnegative and strictly positive n dimensional vectors, respectively. We denote by [0, 1] the n−dimensional unit cube.